Language Models

In statistically based natural language processing a language model is used in the generation of output strings to assess likelihood that a given string of words is a sentence in a particular language. If the information about the ordering of the words in the sentences in a texts were discarded so that each sentence was regarded simply as a bag of words, then the better of two language models would be able to do a better job of restoring their order. According to the simplest view of statistical machine translation, this is just what the language model is called upon to do. For each source sentence, a translation model proposes a bag of words to be used in its translation and it is the job of the language model to order them in the best way. There is, in fact, more to it than this, but it at least gives a setting for the use of language models. Language models are based on so-called n-grams. An n-gram is simply a sequence of n words. The likelihood that a permutation of the given bag of words is a sentence of the language is estimated by taking the product of the probabilities of all the n-grams that it contains. Thus, a sentence of k words contains k n+ 1 n-grams, and the probability of each of them is estimated on the basis of a body of training data. Various tricks are used to allow for permutations that did not occur at all in the training data. The value of n is xed at a fairly low value|say 3, 4, or 5|because, though larger values generally give better results, the amount of training data required to permit a useful estimate of the probability of longer strings grows at an overwhelming rate with value of n. This note results from my re ections on the possibility of language models based on sequences of variable size. If a given set of training data of fairly modest size contains some three or four instances of strings consisting of eight or ten words, then this is surely a remarkable fact, and one from which it should be possible to derive important advantages. Observe, rst of all, that it is a relatively straightforward matter to catalog the repeated sequences of whatever length in a corpus of text. They